Skip to main content

Meeting Prep for 12/18/2023

Flow Process and Design Decisions​

Flow Diagram​

Design Decisions​

Multiple Files​

Reading multiple files.

Instead of needing to know the exact number/name of the files, we are instead placing all the INP files and 'single-column-signal' files in a folder that we generate. This way python can get all inp files in a specified folder rather than requiring us to name each and every file. We will be doing the same thing with the 'single-column-signal' files.

We are also looking into no longer writing out the combined signal file. It is unlikely that anyone will want to look at a combined signal file with thousands of columns. If we modify the combined signal process with the Chama opt code to just pass a large dataframe instead of first writing to a csv then reading the same csv in the next process, it will be more efficient.

Naming​

The current naming conventions are as follows:

Network_RainfallEvent_StartingSubcatchment_WindDirectionDegrees

Instead of using the whole name of the folder for the name in the INP file, we should just use a combination of year and month to cut down on the length of the file name. The year and month combination are unique.

Therefore, 2005Jan02T22 event would become 2005Jan

An example of the file naming would be: Banklick_2005Jan_S326a_90

This would be the INP file name and the signal file name. It would also be the column name for the signal file and the row name in the Chama Scenario File.

Runs in the Super Computer​

We have not made any large runs on the super computer yet. The current count for number of INP files generated in a single ensemble with only 3 variables (Rain event, Start Location and Wind Direction) is 31,590. Since we haven't made any large runs, we don't know when hardware or Chama's software limits will start appearing.

  • Time events: how long does it take to run a single Chama opt simulation, 2, 3, 10... plot the results to extrapolate how long it will take to run more.

Rainfall Data Update​

Caleb sent out 15 folders with Rainfall Data

Caleb_folder

Each folder has a number of files

Caleb_files

  • .hsf file - Hot Start file.
  • .inp file - SWMM Input file.
  • .out file - Binary output file from SWMM.
  • .rpt file - report file from SWMM plain text file.
  • .dat file - rainfall file for the event. (Ask EPA: should we rename this file?)

Adding a Hot-start file to an INP file​

The original Banklick file does not have a hot start. Caleb created the hot-start for us. The hot-start section is added to the FILES section of the INP file.

hot-start save file

This is an image from Caleb's INP file that shows how to save a hot-start file. In our INP files we change the instruction from save to use. So we are not saving a new hot-start file but rather using the one that Caleb sent.

hot-start use file

Question for Caleb:

If we are using the hot-start file but not saving it, do we need to make sure there is rainfall available for the duration of the hot-start?

Duration of the Rainfall​

Rainfall Meta Data

Some of the rainfall events are too large. Particularly 2016Sep13T12 and 2019Jul21T23 as these file are multi-year. There are also files such as 2011Aug07T05 where there is a little over a month of data, but the duration of the event is only 1 day and 14:25 hours.

More questions for Caleb:

  • How much data is needed in our .dat files?
  • Is it more efficient in SWMM to have rainfall files that are close to the start and end dates?
  • If the multi-year data is not an issue and we need to bring in the start and end dates from a third party, such as a CSV file and not the rainfall file, could we just have 1 rainfall file with the data from Jan 2005 - July 2023? Could we store all rainfall in a DB and only grab the data we need per run?

Notes on Banklick​

These are some basic notes on the Banklick network.

  • Number of Junctions in Banklick are 1965
    • We are using each of these junctions as our possible sensor locations.
  • Number of Subcatchments are 351
    • We are using each of these subcatchments as our possible release Locations. (In an ensemble, each location will be used.)
  • Caleb created 15 rain events for the Banklick network.
    • Each Rainfall Event corresponds to a HotStart file.
    • We also have the INP files that Generate the Host-start files and the Output file of a run.
Last updated on by jacobgqc